Robust Action Gap Increasing with Clipped Advantage Learning

Advantage Learning (AL) seeks to increase the action gap between optimal and its competitors, so as improve robustness estimation errors. However, method becomes problematic when induced by approximated value function does not agree with true action. In this paper, we present a novel method, named clipped (clipped AL), address issue. The is inspired our observation that increasing blindly for all given samples while taking their necessities into account could accumulate more errors in performance loss bound, leading slow convergence, avoid that, should adjust advantage adaptively. We show simple AL operator only enjoys fast convergence guarantee but also retains proper gaps, hence achieving good balance large convergence. feasibility effectiveness of proposed are verified empirically on several RL benchmarks promising performance.